CNR Author
operator:
and / or
Typology
operator:
and / or
Language
operator:
and / or
Date
operator:
and / or
Rights
operator:
and / or
2023
Doctoral thesis
Open Access
Orchestration strategies for regression testing of evolving software systems
Merlin Greca R. D.
Context: Software is an important part of modern life, and in most cases, it provides tremendous benefits to society. Unfortunately, software is highly susceptible to faults. Faults are often harmless, but even small errors can cause massive damage depending on the context. Thus, it is crucial for software developers to adopt testing techniques that can help locate faults and guarantee the functionality of both individual components and the system as a whole. Today, there is a trend towards continuously evolving software, in which it is desired that changes such as new features and corrections are delivered to end users as quickly as possible. To ensure correct behavior upon release, development teams rely on regression testing suites, which serve to validate previously-correct features and, when well-designed, avoid the propagation of faults to end users. However, the desire for velocity that comes with continuously evolving software places an additional burden on regression testing practices, as running complete test suites can be a costly process in large-scale software. This challenge has generated a need for novel regression testing techniques, a topic which now enjoys a robust literature within software engineering research. However, there is limited evidence of this research finding its way into practical usage by the software development community; in other words, there is a disconnect between academia and industry on the subject of software testing techniques. Objective: To improve applicability of regression testing research, we must identify what are the main causes of this apparent gap between software engineering academics and practitioners. This is a multifaceted goal, involving an investigation of the literature and of the state of practice. A related goal is to provide examples of test suite orchestration strategies that draw from academic advancements and could provide benefits if implemented on real software. Method: This thesis tackles the aforementioned challenge from multiple directions. It includes a comprehensive systematic literature review covering research published between 2016 and 2022, focusing on papers that bring techniques and discussions that are relevant to the applicability of regression testing research. Along with data extracted from the papers themselves, this discussion of the existing literature includes information received directly from authors through a questionnaire, as well as a survey performed with practitioners, seeking to validate some of the reported findings. Test suite orchestration strategies can be a step towards bridging the so-called industry-academia knowledge gap. To that end we propose a combined approach for regression testing, including techniques extracted from the literature that have promising qualities. This approach is an initial experiment with full test suite orchestration and extended approaches are also discussed. To get a closer understanding of the state of regression testing in a practical sense, a series of interviews were conducted in collaboration with a large technology company. During a seven-week process, we were able to interact with the team and learn the test practices performed on a daily basis and have some insight on the long-term test strategies for the company. The responses of the interviews are reported, edited for readability and confidentiality reasons, and these results are discussed within the larger context of the study. The results from the above components of the studies are then aggregated into two notable outputs. First, a live repository of literature is made available online, containing the current results of the literature review and with the opportunity of expansion as more research is performed in this topic. Then, we provide a list of the most notable challenges for the implementation of regression testing techniques in practice, that were identified during the development of this entire study. Results: This thesis provides the following contributions: a comprehensive literature review of applicable regression testing research; additional context on the literature provided by the authors of cited papers; a preliminary test suite orchestration strategy combining robust techniques from the literature; interviews with practitioners at a major technology company that highlight the challenges faced daily by developers and testers; a live repository of papers to aggregate relevant literature in one online location; a list of challenges that can serve as guidelines for researchers or even as research directions in their own right. Conclusion: There is still much work to be done by the software engineering research and development communities in order to completely close the gap that exists between them. To a great extent, the motivations of researchers and practitioners are not aligned -- while in academia, proposing theoretically sound novel approaches is encouraged to obtain publications, in industry there is a need for techniques that are proven to reduce effort and/or costs. This can only be solved by close collaboration between the two sides, yet a question of who is willing to fund these experiments remain. The data and discussions provided in this thesis show that, although difficult, this is not an impossible problem to solve and there are certain clear steps that can be taken by researchers and practitioners alike to begin addressing it.
Merlin Greca R. D.
Context: Software is an important part of modern life, and in most cases, it provides tremendous benefits to society. Unfortunately, software is highly susceptible to faults. Faults are often harmless, but even small errors can cause massive damage depending on the context. Thus, it is crucial for software developers to adopt testing techniques that can help locate faults and guarantee the functionality of both individual components and the system as a whole. Today, there is a trend towards continuously evolving software, in which it is desired that changes such as new features and corrections are delivered to end users as quickly as possible. To ensure correct behavior upon release, development teams rely on regression testing suites, which serve to validate previously-correct features and, when well-designed, avoid the propagation of faults to end users. However, the desire for velocity that comes with continuously evolving software places an additional burden on regression testing practices, as running complete test suites can be a costly process in large-scale software. This challenge has generated a need for novel regression testing techniques, a topic which now enjoys a robust literature within software engineering research. However, there is limited evidence of this research finding its way into practical usage by the software development community; in other words, there is a disconnect between academia and industry on the subject of software testing techniques. Objective: To improve applicability of regression testing research, we must identify what are the main causes of this apparent gap between software engineering academics and practitioners. This is a multifaceted goal, involving an investigation of the literature and of the state of practice. A related goal is to provide examples of test suite orchestration strategies that draw from academic advancements and could provide benefits if implemented on real software. Method: This thesis tackles the aforementioned challenge from multiple directions. It includes a comprehensive systematic literature review covering research published between 2016 and 2022, focusing on papers that bring techniques and discussions that are relevant to the applicability of regression testing research. Along with data extracted from the papers themselves, this discussion of the existing literature includes information received directly from authors through a questionnaire, as well as a survey performed with practitioners, seeking to validate some of the reported findings. Test suite orchestration strategies can be a step towards bridging the so-called industry-academia knowledge gap. To that end we propose a combined approach for regression testing, including techniques extracted from the literature that have promising qualities. This approach is an initial experiment with full test suite orchestration and extended approaches are also discussed. To get a closer understanding of the state of regression testing in a practical sense, a series of interviews were conducted in collaboration with a large technology company. During a seven-week process, we were able to interact with the team and learn the test practices performed on a daily basis and have some insight on the long-term test strategies for the company. The responses of the interviews are reported, edited for readability and confidentiality reasons, and these results are discussed within the larger context of the study. The results from the above components of the studies are then aggregated into two notable outputs. First, a live repository of literature is made available online, containing the current results of the literature review and with the opportunity of expansion as more research is performed in this topic. Then, we provide a list of the most notable challenges for the implementation of regression testing techniques in practice, that were identified during the development of this entire study. Results: This thesis provides the following contributions: a comprehensive literature review of applicable regression testing research; additional context on the literature provided by the authors of cited papers; a preliminary test suite orchestration strategy combining robust techniques from the literature; interviews with practitioners at a major technology company that highlight the challenges faced daily by developers and testers; a live repository of papers to aggregate relevant literature in one online location; a list of challenges that can serve as guidelines for researchers or even as research directions in their own right. Conclusion: There is still much work to be done by the software engineering research and development communities in order to completely close the gap that exists between them. To a great extent, the motivations of researchers and practitioners are not aligned -- while in academia, proposing theoretically sound novel approaches is encouraged to obtain publications, in industry there is a need for techniques that are proven to reduce effort and/or costs. This can only be solved by close collaboration between the two sides, yet a question of who is willing to fund these experiments remain. The data and discussions provided in this thesis show that, although difficult, this is not an impossible problem to solve and there are certain clear steps that can be taken by researchers and practitioners alike to begin addressing it.
See at:
iris.gssi.it 
| ISTI Repository | CNR ExploRA
2023
Journal article
Open Access
State of practical applicability of regression testing research: a live systematic literature review
Greca R., Miranda B., Bertolino A.
This repository provides a static version of the dataset used for the study"State of Practical Applicability of Regression Testing Research: A Live Systematic Literature Review". The data is available in CSV format (for easy parsing) and HTML format (for readability). HTML files require the included sheet.cssfile (provided by Google Sheets). A non-static version of this data is available via Google Sheets. Furthermore, this data was used to create a live repository of papers on the topic of software regression testing. Itis open for contributions.Source: Computing surveys 55 (2023). doi:10.1145/3579851
DOI: 10.1145/3579851
Metrics:
Greca R., Miranda B., Bertolino A.
This repository provides a static version of the dataset used for the study"State of Practical Applicability of Regression Testing Research: A Live Systematic Literature Review". The data is available in CSV format (for easy parsing) and HTML format (for readability). HTML files require the included sheet.cssfile (provided by Google Sheets). A non-static version of this data is available via Google Sheets. Furthermore, this data was used to create a live repository of papers on the topic of software regression testing. Itis open for contributions.Source: Computing surveys 55 (2023). doi:10.1145/3579851
DOI: 10.1145/3579851
Metrics:
See at:
ISTI Repository | dl.acm.org 
| ACM Computing Surveys | CNR ExploRA
2023
Conference article
Open Access
Orchestration strategies for regression test suites
Greca R., Miranda B., Bertolino A.
Regression testing is widely studied in the literature, although most research on the topic is concerned with improving specific sub-challenges of a wider goal. Test suite orchestration proposes a more comprehensive view of the challenge of regression testing, by merging and combining different techniques with a variety of objectives, including prioritizing, selecting, reducing and amplifying tests, detecting flaky tests and potentially more. This paper presents the key approaches and techniques that form test suite orchestration, along with common evaluation metrics, and discusses how they can be used together to ultimately provide an efficient and effective regression testing strategy. To illustrate the benefits of orchestration, we provide some examples of existing papers that take steps towards this goal, even if the specific terminology is not yet used. Orchestrated strategies utilizing existing regression testing techniques provide a pathway to practicality and real-world usage of the academic literature.Source: AST 2023 - IEEE/ACM International Conference on Automation of Software Test, pp. 163–167, Melbourne, Australia, 15-16/05/2023
DOI: 10.1109/ast58925.2023.00020
Metrics:
Greca R., Miranda B., Bertolino A.
Regression testing is widely studied in the literature, although most research on the topic is concerned with improving specific sub-challenges of a wider goal. Test suite orchestration proposes a more comprehensive view of the challenge of regression testing, by merging and combining different techniques with a variety of objectives, including prioritizing, selecting, reducing and amplifying tests, detecting flaky tests and potentially more. This paper presents the key approaches and techniques that form test suite orchestration, along with common evaluation metrics, and discusses how they can be used together to ultimately provide an efficient and effective regression testing strategy. To illustrate the benefits of orchestration, we provide some examples of existing papers that take steps towards this goal, even if the specific terminology is not yet used. Orchestrated strategies utilizing existing regression testing techniques provide a pathway to practicality and real-world usage of the academic literature.Source: AST 2023 - IEEE/ACM International Conference on Automation of Software Test, pp. 163–167, Melbourne, Australia, 15-16/05/2023
DOI: 10.1109/ast58925.2023.00020
Metrics:
See at:
ISTI Repository | ieeexplore.ieee.org | CNR ExploRA
2020
Conference article
Open Access
Defensive programming for smart home cybersecurity
Rossi M. T., Greca R., Iovino L., Giacinto G., Bertolino A.
Cybersecurity has become a real issue in the development of smart services in the smart home domain, which is formed by a System of Systems where several smart objects are connected to each other and to the Internet. However, these connections expose the devices to possible attackers inside or outside the network, who may exploit software or hardware vulnerabilities to achieve malicious goals. To alleviate this issue, the use of defensive programming assertions can allow the behaviour of smart objects to be monitored and checked for correctness. Furthermore, open source intelligence tools, such as the Shodan search engine, provide features that could be leveraged to detect potential vulnerabilities. In this paper, we propose an approach for the monitoring of Systems of Systems in the smart home domain exploiting the defensive programming paradigm in combination with Shodan APIs.Source: 2020 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW), pp. 600–605, 11/09/2020
DOI: 10.1109/eurospw51379.2020.00087
Metrics:
Rossi M. T., Greca R., Iovino L., Giacinto G., Bertolino A.
Cybersecurity has become a real issue in the development of smart services in the smart home domain, which is formed by a System of Systems where several smart objects are connected to each other and to the Internet. However, these connections expose the devices to possible attackers inside or outside the network, who may exploit software or hardware vulnerabilities to achieve malicious goals. To alleviate this issue, the use of defensive programming assertions can allow the behaviour of smart objects to be monitored and checked for correctness. Furthermore, open source intelligence tools, such as the Shodan search engine, provide features that could be leveraged to detect potential vulnerabilities. In this paper, we propose an approach for the monitoring of Systems of Systems in the smart home domain exploiting the defensive programming paradigm in combination with Shodan APIs.Source: 2020 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW), pp. 600–605, 11/09/2020
DOI: 10.1109/eurospw51379.2020.00087
Metrics:
See at:
ISTI Repository | doi.org | ieeexplore.ieee.org | CNR ExploRA